Emit from latest commit in feature/foundry-release branch:
- Rename job status to 'cancelled' (#42637)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ects/2.2.0

#46536)

* Fix pylint complaining about too many if-else branch in get_openai_client implementation

* Resolved comments

* Remove user-agent construction tests for sync and async clients

* Add telemetry models and update API view properties for enhanced telemetry support

* Enhance sample_session_log_stream.py to verify and assign Azure AI User role for hosted agent identity

* rbac

* Refactor hosted agent samples to streamline session management and RBAC setup

* update rbac util

* tsp loc

* update hosted agent samples

* update recording

* refactor test hosted agents samples to handle RBAC complications

* change log

* samples for async

* Update installation instructions to include aiohttp for async samples

…Azure/azure-sdk-for-python into feature/azure-ai-projects/2.2.0

* Fixed telemetry instrumentor to correctly call is_recording() as a method

* adding tests

* updating changelog

…Azure/azure-sdk-for-python into feature/azure-ai-projects/2.2.0

…s.cmd

…Azure/azure-sdk-for-python into feature/azure-ai-projects/2.2.0

…ects/2.2.0

…ects/2.2.0

* Enhance sample script for Azure AI evaluations

Updated the sample script to support additional command-line arguments for trace evaluations, including agent ID and trace IDs. Modified the lookback hours default value and improved the overall structure for better clarity.

* Add sample for Azure AI Evaluations with hosted agent

This sample demonstrates how to run Azure AI Evaluations against a hosted agent using the azure_ai_target_completions data source, evaluating agents live with built-in quality and safety evaluators.

* Revert TRACE_LOOKBACK_HOURS default to 1 hour to match docstring

Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/f15ae794-b773-4e8c-860b-5aea55873600

Co-authored-by: shrutiyer <9905402+shrutiyer@users.noreply.github.com>

* Add agent eval samples: trace evals and agent-as-target evals

Add two new evaluation samples:

- sample_evaluations_builtin_with_traces.py: Trace-based evaluation with
  three modes (client-side App Insights query, server-side agent ID, and
  explicit trace IDs)
- sample_evaluations_agent_as_target.py: Live agent evaluation using
  azure_ai_target_completions data source

Both samples use the azure_ai_evaluator config pattern with builtin
intent_resolution and task_adherence evaluators.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor evaluator configuration functions

* Delete sdk/evaluation/azure-ai-evaluation/samples/sample_evaluations_agent_as_target.py

* Delete sdk/evaluation/azure-ai-evaluation/samples/sample_evaluations_builtin_with_traces.py

* Add sample_evaluations_agent_as_target to skip list

* Fix pyright error in trace evaluation samples

Use list() to narrow args.trace_ids from Optional[List[str]] to List[str]
inside the elif guard, resolving pyright reportArgumentType error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip sample_evaluations_agent_as_target in test discovery

The sample requires a deployed Azure AI Agent and has no recording,
so it must be excluded from the parametrized test_evaluation_samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor the sample

* pylint fixes

* more pylint fixes

* rename

* rename

* one leftover rename

* undo naming

* fix other samples pylint

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Shruti Iyer <you@example.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…6667)

…nts) (#47009)

…47039)

- Introduced a new sample script `sample_dataset_generation_job_with_evaluation.py` demonstrating end-to-end data generation and evaluation.
- Updated `assets.json` with the new tag for the sample.
- Modified test configuration to include the new sample in the test suite.
- Added header sanitization in `conftest.py` to handle Content-Encoding during tests.

#47058)

* Add samples for Fabric IQ agent with synchronous and asynchronous clients, and update test base with Fabric IQ connection ID

* Refactor instructions in Fabric IQ agent samples for clarity and consistency

* Add 4 data generation job samples to azure-ai-projects

Add four new samples to sdk/ai/azure-ai-projects/samples/datasets/ that
cover the four hero scenarios of the 2.2.0a data generation API
(`pc.beta.datasets.create_generation_job`), plus a README index:

* sample_dataset_generation_job_traces_for_evaluation.py
  Traces -> Evaluation dataset. The Traces source consumes existing
  telemetry, so no model_options are required.

* sample_dataset_generation_job_traces_for_finetuning.py
  Traces -> Supervised fine-tuning JSONL files (train_split=0.8 ->
  two file outputs). Resolves each FileDataGenerationJobOutput's real
  filename via openai_client.files.retrieve, because the service
  returns output.filename=None on FT outputs today.

* sample_dataset_generation_job_simpleqna_with_dataset_source.py
  Multi-source SimpleQnA: a seed Dataset combined with an inline
  Prompt -> Evaluation dataset. Shows that caller-supplied output
  description and tags propagate onto the generated dataset (the
  service also auto-adds a data_generation_job_id tag).

* sample_dataset_generation_job_simpleqna_for_finetuning.py
  Azure OpenAI File source -> short-answer + long-answer fine-tuning
  files (train_split=0.8). Polls the uploaded file until it reaches
  status=processed before submitting the job, otherwise the service
  rejects the reference with `must point to a completed file import`.

All four match the style of sample_dataset_generation_job_with_evaluation.py:
MIT header, parenthesized multi-with, dot-style polling, TERMINAL_STATUSES
set, deletes the data generation job at the end but leaves the generated
output (dataset or files) for the user to inspect. The SimpleQnA samples
also delete the temporary inputs they upload (seed dataset / Azure
OpenAI file).

Each sample preflights `len(output_name) <= 50` because the service
rejects longer output names with a 400 invalidPayload, and uses a short
unique run id suffix so repeated runs do not collide.

All four were live-validated end-to-end against a real Foundry project.

README.md indexes the datasets folder, mirrors evaluations/README.md
structure, and calls out per-sample environment variable requirements
(FOUNDRY_AGENT_NAME for traces samples, FOUNDRY_MODEL_NAME for
simple_qna samples, openai for samples that interact with Azure OpenAI
files).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback on data generation samples

- Add sample_dataset_generation_job_simpleqna_with_agent_source.py:
  self-contained sample that creates a short-lived PromptAgentDefinition,
  uses it as an AgentDataGenerationJobSource for SimpleQnA evaluation, and
  cleans up the agent in a finally block.

- Add sample_dataset_generation_job_simpleqna_with_file_source.py:
  replaces the removed dataset-source sample with a multi-source File +
  Prompt SimpleQnA evaluation job that uploads the seed via the Azure
  OpenAI Files API and verifies description/tag propagation.

- Remove sample_dataset_generation_job_simpleqna_with_dataset_source.py:
  the service no longer supports reading dataset input files for QnA
  generation.

- Rename sample_dataset_generation_job_with_evaluation.py to
  sample_dataset_generation_job_simpleqna_with_prompt_source.py for
  naming consistency with the new _with_<source>_source samples.

- Add full cleanup of generated artifacts (datasets, fine-tuning files,
  prompt agents) across every data-generation sample so repeated runs do
  not accumulate artifacts.

- README: clarify FOUNDRY_MODEL_NAME refers to a model deployment, add a
  link to the Responses-API supported-model list for evaluation jobs and
  a chat-completions hint for fine-tuning jobs, note that traces sources
  also work with third-party (OpenTelemetry-instrumented) agents, and
  document the self-cleaning behavior of the samples.

- CHANGELOG: update reference to renamed sample.

- tests/samples/test_samples.py: update references to renamed sample.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten datasets README: consistency pass

- Bump pip line to azure-ai-projects>=2.2.0 to match all data-gen samples.
- Drop redundant 'matches the FOUNDRY_MODEL_NAME convention' parenthetical
  in the FOUNDRY_MODEL_NAME description.
- Reword the quickstart export line to use a placeholder
  '<your-model-deployment-name>' and rephrase the inline comment.
- Align FOUNDRY_PROJECT_ENDPOINT placeholder casing across the
  prerequisites bullet and the quickstart code block.
- Standardize the agent-source sample reference as 'simple_qna' (snake_case,
  code-formatted) to match the surrounding usage.
- Reorder and rewrite the Data Generation Jobs sample table so rows are
  grouped by scenario (Evaluation first, Supervised fine-tuning second),
  and unify the description voice across rows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Make datasets README more concise

- Replace the three-bullet env-var section with a compact table; drop the
  per-bullet sample-filename enumerations (the Sample Index already maps
  samples to their scenarios) and the agent-source FOUNDRY_AGENT_NAME
  exception note (covered in the sample's own docstring).
- Trim the async/openai prerequisites sentence.
- Collapse the two Data Generation Jobs intro paragraphs into one.
- Trim the file-source row description and drop a redundant 'evaluation'
  qualifier from the traces-eval row (Scenario column already says it).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop redundant 'Running a Sample' section from datasets README

Prerequisites already documents the required environment variables, and
'python <sample>.py' is self-evident; the duplicate snippet only added
noise.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop Data Generation Jobs intro paragraph from datasets README

The table immediately below already shows the source, scenario, and
description for each sample; the prose summary just restated it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop intro blurb from datasets README

The title plus the Sample Index already make the scope obvious.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Smooth out two awkward Sample Index descriptions

- file-source row: drop the 'verifies output metadata propagation' tail
  (it's a test detail, not a user-facing feature) and 'multi-source job
  combining' (the Source column already says File + Prompt).
- agent-source row: replace 'short-lived \PromptAgentDefinition\' with
  'prompt agent' so the description reads naturally without leaking a
  type name into the table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Note third-party agent support in trace sample docstrings

Aligns the trace sample docstrings with the FOUNDRY_AGENT_NAME row in the
datasets README: both Foundry Agents and OpenTelemetry-instrumented
third-party agents are valid sources.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…eval generation (#47057)

* [azure-ai-projects] Add samples for evaluator authoring and adaptive eval generation

Adds five end-to-end samples under sdk/ai/azure-ai-projects/samples/evaluations/
covering the new `beta.evaluators` surface in azure-ai-projects 2.2.0:

* sample_evaluator_rubric_manual.py
  Hand-author a rubric evaluator with create_version + use it in an OpenAI
  evaluation run.

* sample_adaptive_eval_generation_basic.py
  Generate a rubric evaluator from a single Prompt source + use it in an
  OpenAI evaluation run.

* sample_adaptive_eval_generation_all_sources.py
  Demonstrate all four generation source types (Prompt, Agent, Dataset,
  traces). Submits a combined Prompt+Agent+Dataset job and a separate
  traces+Agent job (traces requires a companion source).

* sample_adaptive_eval_generation_iterate.py
  Human-in-the-loop workflow: generate v1, edit dimensions locally
  (boost/drop/add), preserve ALWAYS-ON dimensions verbatim, save as v2.

* sample_adaptive_eval_generation_lifecycle.py
  Full lifecycle: create_generation_job with idempotent operation_id,
  poll, list, delete (with delete_version cascading to the job record).

Style matches sdk/ai/azure-ai-projects/samples/datasets/sample_dataset_generation_job_with_evaluation.py
(DESCRIPTION/USAGE docstring, dot-style polling, module-level constants,
POLL_INTERVAL_SECONDS env var, plain happy-path cleanup at end of `with`).

README.md is updated with a new "Adaptive Eval Generation" section
listing the five samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [azure-ai-projects] Rename adaptive eval samples to rubric evaluator

Drop the "adaptive" terminology and rename the five evaluator samples
under sdk/ai/azure-ai-projects/samples/evaluations/ to use a consistent
"rubric_evaluator" prefix:

  sample_evaluator_rubric_manual.py
      -> sample_rubric_evaluator_manual.py
  sample_adaptive_eval_generation_basic.py
      -> sample_rubric_evaluator_generation_basic.py
  sample_adaptive_eval_generation_all_sources.py
      -> sample_rubric_evaluator_generation_all_sources.py
  sample_adaptive_eval_generation_iterate.py
      -> sample_rubric_evaluator_generation_iterate.py
  sample_adaptive_eval_generation_lifecycle.py
      -> sample_rubric_evaluator_generation_lifecycle.py

Also updates DESCRIPTION/USAGE docstrings, cross-references, internal
operation_id prefixes, and the README section header + table entries
to match the new naming.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback and align with datasets-sample style

slister1001 review:
- Drop 'LLM' / 'deployment' from FOUNDRY_MODEL_NAME docstrings in the four
  generation samples (all_sources / basic / iterate / lifecycle): rubric
  evaluator generation runs inline server side, no project model
  deployment required. basic.py keeps the caveat that the eval-run grader
  still needs a real deployment because it uses the same env var for both
  the generation job and the OpenAI eval run.
- sample_rubric_evaluator_manual.py left untouched: its model is only the
  eval run's judge (genuine deployment).

README:
- Drop the intro paragraph under 'Rubric Evaluators' so the table speaks
  for itself, matching the datasets samples README pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…-S4 (#47034)

* feat(samples): add multi-turn conversation evaluation sample (S1)

Add sample_multiturn_conversation_evaluation.py demonstrating:
- Custom data source config with messages/tool_definitions schema
- Conversation-level evaluators (customer_satisfaction, task_completion, coherence, groundedness)
- Dataset upload and evaluation run with evaluation_level=conversation
- Polling for results

Includes sample JSONL with 3 conversations: basic multi-turn, tool-calling, and extended support dialog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(samples): add S2, S3, S4 multi-turn evaluation samples

- S2: sample_multiturn_trace_evaluation_by_id.py
  Evaluate traces by conversation_id or trace_id
- S3: sample_multiturn_trace_evaluation_agent_filter.py
  Evaluate traces by agent name/version/id with optional smart filtering
- S4: sample_multiturn_conversation_simulation.py
  Simulate multi-turn conversations against an agent and evaluate
- Data: sample_data_simulation_scenarios.jsonl (3 seed scenarios for S4)

All samples use 4 conversation-level evaluators: customer_satisfaction,
task_completion, coherence, groundedness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(samples): narrow dataset.id type for pyright

The .id property on upload_file() returns Optional[str], which pyright
flags when passed to SourceFileID(id=...) which expects str. Split the
chain and add an assert to narrow the type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* update to release version

* update to model

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…iles

* SDK operations for Models, Unit tests and Sample notebook

* modifying sample to .py instead of .ipynb and update changelog

* post emitter fixes and resolving review comments

* fix cpell - azcopy

* re emit from typespec for PendingUploadType changes

* reverting pyproject.toml

* pulling base branch of foundry sdk release for build

* Revert aio _patch.py to base; minimize sync _patch.py diff for BetaModelsOperations

* Address PR #46842 review comments

Darren (dargilco):

- D1: Merge feature/azure-ai-projects/2.2.0 into branch (separate merge commit).

- D2: Remove duplicate '.beta.models' CHANGELOG bullet.

- D3: Remove stray blank line in CHANGELOG.

- D4: Revert apiview-properties.json to base.

- D5: Wire async patched BetaModelsOperations into aio/operations/_patch.py;

  add azure/ai/projects/aio/operations/_patch_models_async.py with async create_version()

  using azure.storage.blob.aio.ContainerClient.

Howie (howieleung):

- H1: Rename helper models_create -> create_version (and validator).

- H2: Rename sample_models{,_async}.py -> sample_models_basic{,_async}.py.

- H3: Print friendly per-field summaries in samples instead of raw model repr.

- H4: Guard against None return from create_version before accessing fields.

Also follows TypeSpec rename: generated create_async() -> pending_create_version().

* Rename .beta.models patched helper create_version -> create

- Renames patched BetaModelsOperations.create_version() to create() (sync + async).

- Renames internal validator _validate_create_version_inputs -> _validate_create_inputs.

- Updates samples, tests, README, and CHANGELOG to use the new name.

- Leaves generated spec method pending_create_version unchanged.

* Add sample recordings for .beta.models and fix generated arg names

- Add parameterized sample tests test_models_samples (sync + async).

- Add slim modelsServicePreparer (only foundry_project_endpoint).

- Add sanitizers for random model name, Foundry storage account/container,

  /workspaces/<name>, azureai:// asset URIs, account/project from FOUNDRY_PROJECT_ENDPOINT,

  and SAS query parameters (sig, skoid, sktid).

- Use per-recording random MODEL_NAME (recsmplmdl<hex>) and sanitize to a stable

  value on playback (Foundry asset-store reserves <name>/<version> permanently).

- Update assets.json tag to point at the new recordings.

Bug fixes surfaced by live runs (renamed generated kwargs):

- pending_upload: body= -> pending_upload_request=

- pending_create_version: body= -> model_version=

- get_credentials: body= -> credential_request=

Applied to patched create() helper (sync + async), all three model samples,

and the test_patch_models mock.

Notes:

- sample_models_basic.py (which uses AzCopy) is excluded from the parameterized

  sample tests because AzCopy traffic isn't captured by the test proxy.

- LLM print-call validation is not invoked for models tests (canary project

  has no Azure OpenAI connection).

* Exclude .beta.models.create from foundry-features header test

The multi-step orchestrator helper performs local input validation before any HTTP call, so the test framework's synthetic placeholder args (e.g. source={}) cause it to raise TypeError before _RequestCaptured is ever raised. Add a shared EXCLUDED_BETA_METHODS mapping and skip excluded methods in both sync and async discovery. The header invariant is still enforced for every underlying HTTP method create() calls (pending_upload, pending_create_version, get).

* Add cspell entries: recsmplmdl, simpleqna, skoid

* Fix pyright and pylint issues in BetaModelsOperations patches

- Replace hasattr-based duck typing with isinstance(dict) check so pyright can narrow types correctly on _extract_pending_upload_targets.
- Remove unused HttpResponseError import.
- Suppress do-not-import-asyncio for asyncio.sleep used in the async polling loop (transport sleep is not applicable).
- Add :param/:keyword/:return/:rtype docstring sections on private helpers to satisfy azure-pylint-guidelines-checker.

* Add @overload variants to .beta.models.create for precise return types

* Rename sample_models_without_patch.py to sample_models_create_and_poll.py

* Remove redundant 'if model is None' check in sample_models_basic.py (covered by overload)

- Removed sample scripts for CRUD operations and upload/download of skills in both synchronous and asynchronous clients.
- Updated `_patch_models.py` to simplify method signatures.
- Adjusted test cases in `test_patch_models.py` for improved readability.
- Updated `foundry_features_header_test_base.py` for consistent formatting.
- Updated `tsp-location.yaml` to point to the latest commit in the Azure REST API specs repository.